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Agenda 


9a 

Welcome  and  Introductions 

P.  Sajda  (Columbia) 

9:15a 

Overview  of  ARO  Decision  &  Neuro-Sciences 

J.  Spoonmore  (ARO) 

9:30a 

Overview  of  ARO/ARL  Projects  at  Columbia 
University 

P.  Sajda  (Columbia) 

9:45a 

ARO  Project:  Cortical  Networks  Underlying 
Rapid  Decision  Making 

P.  Sajda  (Columbia) 

10:30a 

Break 

10:45a 

ARL  CTA  Seedling  Project:  Constructing 
Mutually-derived  Situational  Awareness  via 
EEG-Informed  Graph-based  Transductive 
Inference 

D.  Jangraw  (Columbia) 

1 1 :30a 

Commercialization  and  Operational 

Transition  of  Cortically-coupled  Computer 
Vision  Systems 

P.  Sajda  (Columbia) 

12:00p 

Postdoctoral  exchange  between  ARL  TNB 

J.  Sherwin 

and  Columbia:  Neural  Markers  of  Experts 
and  Novices:  Extensions  to  Gunshot 
Localization 

(Columbia/ARL) 

12:30p 

Lunch  and  Discussion 

All 

2p 

Adjourn 
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Laboratory  for  Intelligent  Imaging  and  Neural  Computing 

LIINC 

...using  principles  of  reverse  engineering  to  characterize  the  cortical  networks 

underlying  rapid  decision  making. . . 


Measuring:  Simultaneous  EEG/fMRI 


Perturbing:  High  Resolution  Informed  TMS 


A 


l-ll 


Early  Visual  Task  Post-Sensory/ 

Perception  Difficulty  Decision  Events 


Time 


Modeling:  Large  Scale  Neural  Simulations 


Decision 


c^H=_x{Vi(t)~VL 
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Building:  Cortically  Coupled  Computer  Vision 


•  •••••  • 
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•  • 

•••  • 
•  • 


MATTERS" 


neurotechnology  research  +  development 
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ARO/ARL  Projects 


•  Cortical  Networks  Underlying  Rapid  Decision  Making  (ARO) 

•  Cortically-Coupled  Computing:  A  Paradigm  for  Mutually- 
Derived  Situational  Awareness  (Seedling  under  ARL  CTA) 

•  Image  Database  and  Neuroimaging  Data  Collection  for  Rapid 
Visual  Decision  Making  (ARL  Technology  Transition  Project) 
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Cortical  Networks  Underlying  Rapid 

Decision  Making 

macro 


single  neurons 


Heekeren  et  a!.,  2004 

functional  imaging 


1.5-2  s 


20%  25%  30%  35%  40%  45% 


%  Phase  Coherence 


from  Philiastides  &  Sajda,  2006 
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Cortical  Networks  Underlying  Rapid 

Decision  Making 


macro 


Heekeren  et  a!.,  2004 


single  neurons 


meso 


functional  imaging 


large  populations 
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Cortical  Networks  Underlying  Rapid 

Decision  Making 

Statement  of  Work 


•  Yearl: 

•  Systematically  design  RSVP  stimulus  presentation  paradigms  and  experiments  for  simultaneous  EEG/fMRI 

•  Run  5  subjects  while  simultaneously  recording  EEG  and  fMRI. 

•  Analyze  results  in  terms  of  spatial  networks  localized  by  integrating  EEG  and  fMRI.  Compare  to  dipole  fits 
and  rLoreta  maps  using  EEG. 

•  Year  2: 

•  Begin  design  of  spatio-temporal  (e.g.  video)  stimulus  experiments  which  investigate  integration  of  evidence 
across  time 

•  Manipulate  difficulty  of  decision 

•  Run  5  subjects  while  simultaneously  recording  EEG  and  fMRI. 

•  Begin  relating  trial-to-trial  variability  to  decision  making  models  such  as  the  drift  diffusion  models  (of  Ratcliff, 
1978,  2009)  and  the  bayesian  decision  models  (of  Beck,  Ma  et  al,  2008.) 

•  Year  3: 

•  Begin  design  of  experiments  for  free-viewing  search  (eye-tracking). 

•  Run  5  subjects  while  simultaneously  recording  EEG  and  fMRI. 

•  Create  a  publicly  available  database  of  neural  signatures  of  trial-to-trial  variability,  code  for  analyzing  EEG / 
fMRI  and  code  for  neurally-informing  decision  models 
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Single-trial  Analysis  of  Simultaneously 
Acquired  fMRI  and  EEG 

. .  .a  window  into  latent  brain  states. . . 
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Why  Acquire  EEG  and  fMRI  Simultaneously? 


•  Electrical  activity  of  the  brain  can  be  correlated  with 
hemodynamic  changes 

•  Understanding  neurovascular  coupling 

•  High  temporal  resolution  of  EEG  complements  high 
spatial  resolution  of  fMRI 

•  fMRI  seeding  of  EEG  source  localization 

•  Correlating  ERPs  (trial-averages)  with  BOLD 

•  Single-trial  variations  in  EEG,  related  to  latent  brain 
states  (attentional  shifting,  cognitive  load,  decision 
evidence,  memory  encoding,  etc.)  can  be  correlated 
with  hemodynamic  activity. 
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Challenges 


complicated  equipment 


Amp  Signal  Out  cable 


auditory  noise 


Other  devices  not  shown: 

-  goggles 

-  headphones 


3T  Scanner  Room 


EEG  Cart  f  Other  connections  to  cart: 
-scanner  trigger 

-  video  to  goggles 

-  audio  to  headphones 


3T  Control  Room 


bias  field 


gradient  artifact 


IHHRR 

111 

III 

III 

01  23456789  10 


BCG  artifact 


How  to  combine  the  data? 
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Our  Solutions 


Mpu 


EEG+bcg 

BCG 


Using  EEG  Single-Trial  Variability  to  Construct  fMRI  Regressors 

Stand  Targ  RT  Stand  Stand  Targ  RT 
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Observing  latent  trial-to-trial  fluctuations  of  attention/ 
alertness/perceived-stimulus-salience 


Discriminating  Component  (yT) 

350  ms  component 


10 
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200  ms  component 


Discriminating  components  and  fMRI  regressor  fits  for  a  single  target  trial 
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EEG-derived  BOLD  model  across  all  trials 
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Observing  latent  trial-to-trial  fluctuations  of  attention/ 
alertness/perceived-stimulus-salience 


Modulation  of  attention/alertness 
via  the  locus  coeruleus  (LC)? 


LC 


•  Evidence  that  low  freq  LFPs  (<  30Hz)  have  neg.  corr. 
with  BOLD  (Mukamel  et  al,  2005) 

•  LC  associated  with  P300  (Aston-Jones  &  Cohn,  2005) 

•  LC  fires  phasically  close  to  response  and  appears  to 
modulate  decision/response  and  not  sensory  inputs 
(Clayton,  et  al.  2004) 
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Neurally-lnformed  Models  of 
Decision  Making 


RTdistribution  for  correct 
responses 

Face”  Response 

Variability  in 
drift  rate  across 
trials  (drift~N(v,r|)) 

Time - ► 

Variability  within  trials  (SD=s) 

-Car”  Response 


RT  distribution  for  errors 


TAG 

(Transductive  Annotation  by  Graph) 
Semi-supervised  machine  learning 


level  of  abstraction 
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Model  Summary 


Macaque  VI,  input  layer  (4Ca/|3) 

4  ocular  dominance  columns 
64  orientation  hypercolumns 
16  mm2  cortical  area,  0-10°  eccentricity 
Approx.  65,000  neurons  (integrate-and-fire)  per 
configuration  (a,  p,  0°  ’10°)  =  260,000  neurons. 

Cell  populations: 

•  75%  excitatory  cells 

•  25%  inhibitory  cells 

•  30%  receive  LGN  input 

•  70%  do  not  receive  LGN  input 

Anatomically  realistic  LGN  input  and  retinotopic 
map 

Anatomically  “correct”  cortical  length  scales: 

•  rEaxon=200[im,  rlaxon=100(im 

•  rEdend=50|im,  rldend=50^im 
First-order  LGN  temporal  kernel 
Cortical  time  scales: 

•  AMPA  (5  msec) 

•  NMDA  (50  msec) 

•  GABA  (10;  100  msec) 


(A) 


VISUAL  FIELD  CORTEX 
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Model  Has  Realistic  Response  Properties 


receptive  fields 


Wielaard  and  Sajda,  2006 


orientation  pinwheels 


characteristics  of  S  &  C  cells 


1  - LGN  input 

‘  .  no  LGN  input  i* 


F1/F0 
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Realistic  Nonclassical  Response  Properties 

. .  .suppression  and  RF  shifts  at  high  and  low  contrasts. . . 


Experimental  Data 


Model 


(A)  (B) 


(C)  (D) 


from  Sceniak  et.  al.,  1999 


(Wielaard  and  Sajda,  2006) 
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The  Perceptual  Decision  Making  Task 


20%  25%  30%  35%  40% 


45% 
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Mapping  A  Rapidly  Flashed 
Static  Scene  into  Spikes 


Firing  Rate  (Hz)  Firing  Rate  (Hz) 
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Using  Population  LFPs  for  Discrimination 

(a)  (b) 

Face  Car 


Face-Car 


50  100  150  200  250 

Time  (ms) 


Discrimination 
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Sparseness  of  VI  Model  Activity 
in  Space  and  Time 

(a)  (b) 


Position  X 


4000 r 
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g  2000 I 


10001 


kurtosis  =  10.26 


50  100 
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The  Spike-based  Feature  Space 
or  “Neural  Word” 
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Decoding  High-dimensional 
Spatio-temporal  Dynamics 


time 
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Time  Bin 
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Training  and  Testing  Procedure 


•  Used  12  face  images  and  12  car  images 

•  Divided  into  training  and  testing 

•  6  completely  different  faces  and  cars  used  to  train  and  test 

•  Manipulated  phase  coherence  of  images  at  each  of  7  different 
coherence  levels. 

•  Images  repeated  30  times  resulting  in  a  total  of  5040  trials  (2520 
for  training  and  2520  for  testing). 

•  Images  covered  approximately  4  degrees  of  visual  angle  for 
simulations. 
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Discrimination  Results 


Neurometric  curve  NOT  optimized  to 
match  psychometric  curve! 


Average  across  6  models  and  10  subjects 


Matrix  word  is  better  match  to  psychometric  results. 

High  temporal  precision  (temporal  code)  is  superior  to  rate  code. 
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Analysis  of  Informative  Dimensions 


(a)  #  Informative  Dimensions  (b)  #  Informative  Neurons  (c)  Difference 


More  neurons  are  recruited  at  lower  coherences 
Few  neurons  are  utilized  at  more  than  one  time  bin 
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Selected  Neurons 


Coherence  25% 


Coherence  45% 


(b) 


20 
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20  40  60 
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Uniqueness  of  Informative  Dimensions 
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Where  We  Are: 

Task:  3-Choice  Perceptual  Decision 


Face/House 


Face/House 


Car/House 


(a) 

Car/House 


Face/Car 


RTdistribution  for  correct 
responses 

“Face”  Response 

Variability  in 
drift  rate  across 
trials  (drift~N(v,r|)) 

Time - ► 

Variability  within  trials  ( SD=s ) 

-Car”  Response 


RT  distribution  for  errors 
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